Repository navigation
package: Add missing openssl and decompression utilities to KVM agent dependencies - #14307
weizhouapache wants to merge 1 commit into
Conversation
|
@blueorangutan package |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #14307 +/- ##
=========================================
Coverage 20.21% 20.21%
- Complexity 20747 20750 +3
=========================================
Files 6426 6426
Lines 580181 580181
Branches 71033 71033
=========================================
+ Hits 117282 117288 +6
+ Misses 450184 450180 -4
+ Partials 12715 12713 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Match the KVM agent dependencies added in apache/cloudstack#14307: openssl, bzip2, gzip and unzip are needed for template decompression (.bz2/.gz/.zip) and direct download handling. Install them via apt on Debian/Ubuntu and dnf on CentOS 8/EL9.
… dependencies Add bzip2, gzip, unzip, and openssl as explicit dependencies for the KVM agent. These utilities are required for: - Template decompression (bunzip2 for .bz2 files, gzip for .gz, unzip for .zip) - SSL/TLS operations - Direct download template handling on Ceph/RBD storage Fixes direct download failures when template files are compressed (e.g. .qcow2.bz2).
7d69608 to
265fe9d
Compare
|
@blueorangutan package |
|
@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
There was a problem hiding this comment.
🟡 Changes recommended
The agent still lacks explicit file and xz dependencies required by its supported extraction path.
3 open findings
What changed in this PR
Adds explicit SSL and decompression runtime dependencies to KVM agent packages.
Changes:
- Adds OpenSSL, bzip2, gzip, and unzip dependencies across supported distributions.
| File | Description |
|---|---|
packaging/suse15/cloud.spec |
Updates SUSE agent dependencies. |
packaging/el8/cloud.spec |
Updates EL8 agent dependencies. |
debian/control |
Updates Debian agent dependencies. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| Package: cloudstack-agent | ||
| Architecture: all | ||
| Depends: ${python:Depends}, ${python3:Depends}, openjdk-17-jre-headless | java17-runtime-headless | java17-runtime | zulu-17, cloudstack-common (= ${source:Version}), lsb-base (>= 9), openssh-client, qemu-kvm (>= 2.5) | qemu-system-x86 (>= 5.2), libvirt-bin (>= 1.3) | libvirt-daemon-system (>= 3.0), iproute2, ebtables, vlan, ipset, python3-libvirt, ethtool, iptables, cryptsetup, rng-tools, rsync, ovmf, swtpm, lsb-release, ufw, apparmor, cpu-checker, libvirt-daemon-driver-storage-rbd, sysstat, python3-libnbd, socat | ||
| Depends: ${python:Depends}, ${python3:Depends}, openjdk-17-jre-headless | java17-runtime-headless | java17-runtime | zulu-17, cloudstack-common (= ${source:Version}), lsb-base (>= 9), openssh-client, qemu-kvm (>= 2.5) | qemu-system-x86 (>= 5.2), libvirt-bin (>= 1.3) | libvirt-daemon-system (>= 3.0), iproute2, ebtables, vlan, ipset, python3-libvirt, ethtool, iptables, cryptsetup, rng-tools, rsync, ovmf, swtpm, lsb-release, ufw, apparmor, cpu-checker, libvirt-daemon-driver-storage-rbd, sysstat, python3-libnbd, socat, openssl, bzip2, gzip, unzip |
| Requires: openssl | ||
| Requires: bzip2 | ||
| Requires: gzip | ||
| Requires: unzip |
| Requires: openssl | ||
| Requires: bzip2 | ||
| Requires: gzip | ||
| Requires: unzip |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19489 |
| Requires: openssl | ||
| Requires: bzip2 | ||
| Requires: gzip | ||
| Requires: unzip |
There was a problem hiding this comment.
should we add xz here too? the direct download code also unpacks .xz templates, so those would still fail
| Package: cloudstack-agent | ||
| Architecture: all | ||
| Depends: ${python:Depends}, ${python3:Depends}, openjdk-17-jre-headless | java17-runtime-headless | java17-runtime | zulu-17, cloudstack-common (= ${source:Version}), lsb-base (>= 9), openssh-client, qemu-kvm (>= 2.5) | qemu-system-x86 (>= 5.2), libvirt-bin (>= 1.3) | libvirt-daemon-system (>= 3.0), iproute2, ebtables, vlan, ipset, python3-libvirt, ethtool, iptables, cryptsetup, rng-tools, rsync, ovmf, swtpm, lsb-release, ufw, apparmor, cpu-checker, libvirt-daemon-driver-storage-rbd, sysstat, python3-libnbd, socat | ||
| Depends: ${python:Depends}, ${python3:Depends}, openjdk-17-jre-headless | java17-runtime-headless | java17-runtime | zulu-17, cloudstack-common (= ${source:Version}), lsb-base (>= 9), openssh-client, qemu-kvm (>= 2.5) | qemu-system-x86 (>= 5.2), libvirt-bin (>= 1.3) | libvirt-daemon-system (>= 3.0), iproute2, ebtables, vlan, ipset, python3-libvirt, ethtool, iptables, cryptsetup, rng-tools, rsync, ovmf, swtpm, lsb-release, ufw, apparmor, cpu-checker, libvirt-daemon-driver-storage-rbd, sysstat, python3-libnbd, socat, openssl, bzip2, gzip, unzip |
There was a problem hiding this comment.
does the agent also need the file package? it uses that to tell if a downloaded template is compressed at all
…ide NFS and iSCSI (#237) * Add Ceph (RBD) as a primary storage option alongside NFS and iSCSI Ceph nodes are provisioned out-of-band by a chained "ceph-deployer" Jenkins job; this adds the Ansible side: a cloudstack-cephstorage role that carves two 30GB RBD pools (with per-pool cephx credentials) out of the cluster, registers them with CloudStack via cloudmonkey, and points the generated Marvin config at the RBD pools instead of NFS. NFS mount points are still created but no longer added to CloudStack when env_priprot=ceph. * Fix Jenkins failure: Ceph group_vars templating relied on vars only in group_vars/all group_vars/all is gitignored, so the Jenkins box's copy predates the new def_env_ceph_*/def_pri_password_ceph defaults. Referencing an undefined def_ var inside default() aborts the whole Jinja render, leaving a stale unrendered group_vars file and tripping the "curly braces found" check. Chain a literal fallback so these vars work even before the production group_vars/all is updated. Also guard the new primary_storage_hosts_ceph inventory group so it doesn't emit blank-hostname entries on non-Ceph runs where no Ceph IPs are supplied. * Ceph primary storage: SSH to Ceph nodes with the same password as other storage hosts The cloudstack-cephstorage role was failing SSH auth against the Ceph nodes because pri_password_ceph fell back to an empty-string default (def_pri_password_ceph isn't set on existing group_vars/all files). Drop the separate Ceph-only SSH credential and just reuse pri_password/def_pri_password, matching how the other primary storage hosts authenticate. * Fix Ceph RBD storage pool URL using wrong cephx username addpools.yml creates a cephx identity scoped to each pool (client.<poolname>), but the rbd:// URLs built for cloudmonkey and the Marvin config hardcoded env_ceph_username ("admin") as the username while supplying that pool-scoped key - a cephx identity/key mismatch that would fail authentication regardless of any client library version issue. Use the pool name as the username, matching the identity addpools.yml actually created, and drop the now-unused env_ceph_username/def_env_ceph_username plumbing. * Use pri to size the number of Ceph RBD pools instead of env_ceph_poolcount pri already controls how many primary storage pools get created for NFS/iSCSI/ESXi/XenServer/KVM - reuse it for Ceph too instead of a separate env_ceph_poolcount knob, so one extra-var consistently controls pool count across all primary storage protocols. * Fix Marvin test_data.py: internal nfs/nfs2 pools always use NFS, not env_priprot marvin_pri1/nfs2 are Marvin's own internal test primary storage pools - plain NFS directories always created on env_prihost by cloudstack-nfsshares regardless of the zone's primary storage protocol. The template interpolated env_priprot for their URL scheme, so with env_priprot=ceph the generated test_data.py claimed "Ceph://..." for directories that are actually NFS exports. Hardcode nfs:// for these two. * Install ceph-common on KVM hosts for Ceph RBD volume encryption support CloudStack's RBD volume encryption on KVM needs the rbd CLI/config from ceph-common on the hypervisor. Install it via the generic package module (works across all supported distros) when env_priprot=ceph. * marvin: fetch ipmisim.py fix from GitHub to patch installed package Replace the old hardcoded wget-based fix (which assumed a fixed python3.6 site-packages path) with a pip-show-based lookup of the installed ipmisim package location, then fetch the updated ipmisim.py directly via get_url from shapeblue/ipmisim@a8e56bd (tip of shapeblue/ipmisim#18), so it works regardless of Python version/site-packages layout and doesn't need to be vendored in this repo. * Move ceph-common install into per-distro KVM task files The generic package: module task in roles/kvm/tasks/main.yml wasn't valid there. Install ceph-common via the distro's own module instead: apt in ubuntu.yml/debian.yml, dnf in centos8.yml and el9.yml, each gated on env_priprot=ceph. * Do not percent-encode the Ceph RBD secret in storage pool URLs Traced both the cloudstack-cloudmonkey and apache/cloudstack source: cloudmonkey's encodeRequestParams() already percent-encodes the whole url= value exactly once via Go's url.QueryEscape before sending the API request, and CloudStack's UriUtils.getRbdUrlInfo() expects to receive the raw literal secret (it does its own "+"->"-" and "/"->"_" substitution before handing it to a lenient base64 decoder that accepts either alphabet). Pre-encoding the secret ourselves with Jinja's urlencode double-encoded it, leaving literal %2B/%2F/%3D text embedded in the credentials instead of the real characters. Pass the raw secret as-is. * Add SharedMountPoint primary storage support (env_priprot=sharedmountpoint) NFS shares are created on the primary storage host as for NFS, but are not registered as NFS pools. The new cloudstack-sharedmountpoint role mounts each KVM share on every KVM host at the same local path (env_prismp_path, default /mnt/sharedmp/), and the pools are added to CloudStack and the Marvin config as SharedMountPoint primary storage. Teardown unmounts them before the NFS shares are removed. * Install openssl and decompression utilities on KVM hosts Match the KVM agent dependencies added in apache/cloudstack#14307: openssl, bzip2, gzip and unzip are needed for template decompression (.bz2/.gz/.zip) and direct download handling. Install them via apt on Debian/Ubuntu and dnf on CentOS 8/EL9. * removeproject: skip project deletion when no matching project is found cmk deleteproject id= cleanup=true fails with "Invalid value provided for API arg: id" when the jq lookup finds no project matching build_project (e.g. already removed in a prior partial run). Only run the delete when a projectid was actually found. * Drop removeprimary for Ceph and SharedMountPoint storage The Ceph cluster and its RBD pools are torn down by the separate ceph-deployer Jenkins job, so the cloudstack-cephstorage role (and the destroyvms.yml play using it) no longer needs a removeprimary action. SharedMountPoint only reuses the NFS shares, which are still removed by the cloudstack-nfsshares removeprimary step, so the cloudstack-sharedmountpoint role does not need one either.

Description
Add bzip2, gzip, unzip, and openssl as explicit dependencies for the KVM agent. These utilities are required for:
Fixes direct download failures when template files are compressed (e.g. .qcow2.bz2).
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
How did you try to break this feature and the system with this change?